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Abstract 

English: 

Literacy in the United States is a widespread problem. There are many Americans who do not have the ability to 
read at a level above the ninth grade. Today’s technological advances have resulted in more information being 
placed on the Internet. Consequently, the same individuals who had a limited capacity to read information from 
traditional sources will have similar difficulties reading information on the Internet. This study examined three well- 
known Internet sites for readability. Results of the study indicated that the average reading level for all three sites 
exceeded the level intended for public consumption of written information. Results of the study indicate that health 
educators need to be aware of the reading levels of the health information they are placing on the Internet. If the 
level of health information material is above the comprehension level of the general public, many individuals will be 
at a disadvantage in comprehending health information required to make personal health decisions. 

Spanish: 

El analfabetismo en los Estado Unidos de Norteamerica es un problema amplio. Existen muchos estadounidenses 
que no tienen la habilidad de leer a un nivel sobre el tercer aho de escuela secundaria o bachiller (9° grado). Los 
avances tecnologicos de hoy en dia han resultado en que mas informacion se encuentre disponible en el Internet. 
Como consecuencia, los mismos individuos que tienen capacidad limitada para leer informacion de maneras 
tradicionales tendrcin dificultad leyendo la informacion en el Internet. Este estudio examind tres pdginas web 
conocidas, en cuanto a su nivel de lectura y entendimiento. Resultados del estudio indican que el nivel de lectura 
promedio para los tres sitios excede el nivel intencionado de informacion escrita para el publico. Resultados del 
estudio indican que educadores para la salud necesitan estar conscientes de los niveles de lectura y entendimiento 
de la informacion en salud que estdn presentando en el Internet. Si el nivel de la informacion en los materiales de 
salud se encuentra sobre los niveles de entendimiento del publico en general, muchos individuos se encontrardn en 
desventaja de comprension de informacion de salud requerida para realizar decisiones. 
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Introduction 

In the United States there are millions of individuals 
who do not have the ability to read above the ninth 
grade level. The inability to read and comprehend 
written forms of communication such as magazines, 
books, brochures or informational pamphlets, places 
them at a disadvantage in acquiring important health 
information. Recently, there have been research 
studies that focused on specific patient populations to 
determine their reading abilities. Investigating 
research on English-speaking diabetic patients found 
that while 60% could understand information written 
at the sixth grade level, only 21% could understand 
information written at the ninth grade level (Berland, 
et al., 2001). Other studies have found a median 
reading level of ninth to tenth grade in emergency 
department patients (Baker, et al., 1996) and a 
median reading level of seventh to eighth grade in 


cancer patients, patients in urban clinics, and parents 
of pediatric patients at a University Hospital 
(Berland, et al., 2001). 

Today, more health agencies are relying on the 
Internet to disseminate health information. Although 
the Internet may provide convenient access to health 
education materials for some patients, it does add 
another level of complexity for others. Many patients 
will first have to overcome the obstacle presented by 
the technology. Once those patients are able to 
understand, utilize and access the technology of the 
Internet, their low level of reading comprehension 
may still present a barrier. 

Many of the Internet sites contain personal 
testimony and are easier to read and understand than 
are the more reliable sources such as those of non- 
profit organizations, health foundations, and 
government agencies. Consequently, the individuals 
who have a limited capacity to read information from 
traditional sources will have similar problems reading 
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and comprehending reliable health information on 
digital sources. 

When individuals have limited reading abilities 
or read at the minimum standard for the population 
(ninth grade level), they will fall short of accessing 
and interpreting information required to make 
successful health decisions. With the growth of the 
Internet, and the development of more sophisticated 
search tools, the opportunity to obtain health 
information via this medium has grown 

tremendously. Rosch (1999) reported that within the 
popidation seeking health information on the Internet, 
81 percent reported finding the health information to 
be useful. Most of these individuals were looking for 
specific information on a particular disease condition, 
educational services, medications, physical fitness 
and alternative treatments. 

Previous research has been published regarding 
quality of Internet sites. Tillman (1997) wrote about 
evaluation criteria of Internet sites in terms of 
authorship, content, credibility and validity of 
information. Rippen (2000) specifically explored the 
standards of evaluative criteria when looking at 
health information on the Internet. They found 
reliable health information sites are typically 
associated with government sources or from 
organizations/foundations that are closely affiliated 
with government sources. They also reported that 
quality of health information on the Internet could 
affect lives more so than printed media because it is 
readily available and can reach the health consumer 
within seconds. Without the ability to fully 
comprehend written information, an individual may 
make decisions based on limited knowledge. 

Despite the identification of a ninth grade 
reading level, most health information is written over 
the population mean (French and Larrabee, 1999). 
According to Hart (1993), about two in five 
Americans have completed high school and never 
went to college, and one in five have never 
completed high school. The significance of these 
figures is important with regards to reading ability. 
Approximately 60 percent of the people in the United 
States will require maximum effort to comprehend 
reading material which scores over the 12 th grade 
level on a standard readability test. Reading experts 
report that the majority of the population prefers to 
read written material three grade levels under their 
actual education level. Standard publications, 
regardless of source, would need to be at the ninth 
grade level for maximum comprehension. Johnson 
(1992), performed a study, which assessed the 
readability of health education pamphlets, and 
booklets, which were distributed among clients at 
three local health departments in North Carolina. The 
results of their study indicated that only 15.4 percent 


of all materials received from commercial and non- 
for-profit vendors were at an acceptable reading 
level. They identified acceptable levels as being 
between the fifth and seventh grades. 

MacColl (1998), reported that most health 
information located on the Internet is reliable and 
beneficial to the health consumer. Many sites report 
the results of clinical trials or experimental research. 
The content of this type of written material is 
beneficial to the health consumer looking for 
information on a specific health condition, disease or 
dysfunction. However, the typical health consumer 
cannot comprehend most of these research pieces 
because they are filled with technical terminology. 
The problem is then magnified because most readers 
cannot derive meaning from statistical techniques 
utilized in reporting research findings. 

Readability formulas have been used for years to 
establish if the intended audience for a particular text 
can comprehend the message or knowledge intended. 
A readability formula is a mathematical calculation 
derived by regression analysis. Readability formulas 
are commonly used to calculate reading levels. 
Freimuth (1979), in his investigation of reading 
formulas, suggests that the selection of reading 
formula is dependent on intended usage. Reading 
scales with high predictive validity and intended for 
general audiences require two variables, a word 
variable and a sentence variable. The Flesch-Kinkaid 
and the SMOG Formulas are examples of the two- 
variable formulas. The McLaughlin SMOG 
Readability Formula is commonly used because of its 
ease in calculation and accuracy. It is based 100 
percent on reading comprehension. The SMOG 
Formula consists of counting sentences in a text from 
the beginning, middle and end of the written material. 
Words with more than three syllables are counted in 
each of the ten sentences. The polysyllabic words are 
then added and the square root is calculated. The last 
portion of the formula requires the evaluator to add 
three to obtain the final reading level (French and 
Larrabee, 1999). 

The Flesch-Kinkaid readability formula produces 
a score equivalent to education grade levels. A score 
of twelve is equivalent to a twelfth grade reading 
level. The formula is an objective quantitative tool 
that measures number of words per sentence, 
sentences per paragraph, and length of sentences and 
polysyllabic words. 

This study examined whether there was a 
difference in readability of health information on the 
Internet by examination of three health web sites that 
have been identified as offering reliable information: 
healthline.com, healthfmder.gov, and DrKoop.com. 
The answers to the following questions were sought: 
(1) What are the differences between 
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healthfmder.gov, healthline.com, and DrKoop.com 
web sites with regard to readability of health 

information on the Internet using the Flesch-Kincaid 
Reading Scale? (2) What are the differences between 
healthfmder.gov, healthline.com, and DrKoop.com 
web sites with regard to readability of health 

information on the Internet using the SMOG Reading 
Formula? (3) What are differences between the 
SMOG formula and the Flesch-Kinkaid reading scale 
when evaluating health information sites on the 
Internet? 

Methods 

The sample for this study consisted of readability 
levels from: healthfmder.gov, healthline.com, and 
DrKoop.com. Data was collected on two dates one 
month apart to allow the feature articles to change for 
healthline.com. and DrKoop.com. For each of these 
sites the first five feature articles were selected from 
each month. The sample selection for 

healthfmder.gov was completed in one collection. 
Healthfmder.gov does not offer feature articles but 
has specific hot topic areas. The fifth and tenth hot 
topic areas were selected. Under each of these topic 
areas, the first five articles were selected. All articles 
included in the study from healthline.com , 
DrKoop.com, and healthfmder.gov were copied from 
the Internet and put into a word file. A Flesch- 
Kinkaid was performed on each of the 30 articles 
using the standard software on a Microsoft Word® 
program to determine reading level. The SMOG 
formula was manually calculated on each of the 30 
articles. 

Analysis of Data 

The following analyses were performed to 
determine the differences between Internet sites on 
healthline.com, DrKoop.com, and healthfmder.gov in 
regards to readability. Measures of central tendency 
were calculated for each of the Internet sites using the 
Flesch-Kinkaid Reading Scale and the SMOG 
Reading Formula. ANOVA was used to examine if 
healthline.com, DrKoop.com, and healthfmder.gov 
Internet sites differed using the Flesh-Kinkaid 
Reading Scale (Research Question #1). ANOVA was 
also used to examine ifhealthline.com, DrKoop.com, 
and healthfmder.gov Internet sites differed using the 
SMOG Reading Formula (Research Question #2). T- 
tests were calculated to determine if the Flesch- 
Kinkaid Reading Scale and the SMOG Reading 
Formula were equal measures of readability 
(Research Question #3). 

Results 

This study sought to answer three research questions: 
1) Is there a difference between healthfmder.gov, 
healthline.com, and DrKoop.com Internet sites with 
regard to readability of health information using the 


Flesch-Kincaid Reading Scale? 2) Is there a 
difference between healthline.com, healthfmder.gov 
and DrKoop.com Internet sites with regard to 
readability using the SMOG Reading Formula? 3) Is 
there a difference between the SMOG formula and 
the Flesch-Kinkaid reading scale when evaluating 
health information sites on the Internet? An ANOVA 
was conducted to determine the difference of the 
Internet sites using the Flesch-Kincaid Reading 
Scale. The F statistic was significant: F=9.17 (Fcv, 
.05=3.35, Fcv, .01=5.49), R2 =.40 (Table 1). The 
results of the F statistic indicate that there was a 
difference among the three sites regarding readability 
when using the Flesch-Kincaid Scale (Table 1). 


Table 1. Analysis of Variance for SMOG 


Source 

DF 

SS 

MS 

Between Group 

2 

60.88 

30.438 

Within Group 

27 

89.62 

3.319 

Total 

29 

150.49 



Note: Readability of Internet Sites Using the SMOG 
Formula: Fcv., 05=3.35, Fcv., 01=5. 49 

An ANOVA was conducted to determine the 
difference of the Internet sites using the SMOG 
Formula. The F statistic was significant F=5.37 at the 
.05 level (Fcv, .05=3.35), but not significant at the 
.01 level (Fcv, .01=5.49), R2 =28 (Table 2). The 
results of the F statistic indicated that there was a 
difference between the sites using the SMOG formula 
when tested at the .05 level of significance; however, 
no significant difference at the .01 level (Table 2). 


Table 2. Analysis of Variance for Flesch- 
Kincaid Reading Scale 


Source 

DF 

SS 

MS 

F 

Between Group 

2 

33.32 

16.66 

5.37 

Within Group 

27 

83.769 

3.10 

Total 

29 

117.09 




Note: Readability of Internet Sites Using the 
Flesch-Kincaid Reading Scale: Fcv., 05=3. 35 
Fcv. 01=5. 49 


An independent one-tailed t-test was conducted 
to determine if there was a difference between the 
Flesch-Kincaid Reading Scale and the SMOG 
Formula. The t-test was not significant t(30)=.54 
p=.05. The results of the t-test indicate there is no 
difference between the Flesch-Kincaid Reading Scale 
and the SMOG Formula when evaluating health 
information on the Internet. 

Additional findings through statistical 
calculations indicated the mean reading levels for 
sites using the Flesch-Kincaid Reading Scale were 
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healthline.com, mean =11.92, healthfmder.gov, mean 
= 9.53, and DrKoop.com, mean = 1 1.6. The mean for 
reading levels using the SMOG Formula were 
healthline.com, mean = 14.48, healthfmder.gov, 
mean = 1 1.46, and DrKoop.com, mean = 12.33. 
Summary 

The majority of studies regarding health 
information have dealt with evaluating reliability 
(Silberg, W. M., Lundberg, G. D., & Musacchio, R. 
A., 1999; Bower, H., 1996). Fewer studies have been 
conducted on the readability of health information on 
the Internet. Although literacy has been a problem in 
the United States for many years with other forms of 
written materials, the Internet has received little 
attention. 

The statistical analysis revealed a significant 
difference in the sites when an ANOVA was applied 
for each reading instrument. An independent t-test 
revealed that the two reading scales the Flesch- 
Kincaid and the SMOG had no significant difference 
between the scales. 

Conclusions 

The results of this study revealed a difference among 
healthline.com, healthfmder.gov, and DrKoop.com 
when establishing reading level using both the 
Flesch-Kincaid Reading Scale and SMOG Formula. 
There was no significant difference in the 
identification of establishing reading levels of written 
material on the Internet between the Flesch-Kincaid 
and SMOG Formula. 

Discussion 

The American population has struggled with the 
problem of literacy for many years. A large 
proportion of the population has only basic literacy 
skills (Jenner, 1994). The deficit of reading skills will 
be similar regardless of the medium of written 
communication. With the growth of the Internet, the 
American population will see a dramatic shift toward 
electronic sources of information as opposed to 
traditional forms in papers, books and magazines. 
The equipment and operational knowledge required 
for Internet access, coupled with readability issues, 
will magnify the existing problem of making health 
care information accessible and comprehendible. 

There was a difference identified among the 
three sites examined with regards to reading level 
using the Flesch-Kincaid Reading Scale and the 
SMOG Formula. Healthfmder.gov had the lowest 
mean reading level among the three sites evaluated. 
Healthline.com and DrKoop.com had very similar 
means. The results of the reading levels for all sites 
were above the recommended reading level (ninth 
grade) of written communication recommended for 
public consumption. This indicates that the health 


information on the Internet well exceeds the average 
readability level of most individuals. 

Efforts to ensure that health information sources 
are both reliable and readable have been limited 
(Gottlieb & Lingle, 1997; Gorman, 2000). The results 
of this study revealed that reputable government 
sources fail to meet the readability standards for the 
majority of individuals. Therefore, the researchers 
conclude that reliability and readability of health 
information does not necessarily occur 
simultaneously. 

Recommendations for the Health Educator 

Based on the results of this study several 
implications for the health educator should be noted. 
Individuals seeking health information must be able 
to read and understand the health sites from which 
they seek health information. If they do not possess 
the necessary skills for comprehension, they will be 
at a disadvantage when attempting to make informed 
choices. Health educators posting information on the 
Internet need to be aware of the reading level of the 
message they are trying to convey and target their 
information to a particular audience. It is also 
important for health educators to remember that 
reliable health information typically has a higher 
reading level, because it is tilled with medical terms 
regarding health, disease, and dysfunction of the 
human body. A final recommendation would be for 
further studies to investigate the accessibility of 
individuals with disabilities to the Internet. If 
individuals with disabilities are not able to physically 
access health related materials via Internet 
technology, they will be at a disadvantage when 
attempting to make educated health care decisions. 

The Internet has become one of the largest 
suppliers of health information. The fact that over 
100 million Americans will go online in search of 
health-related information, and that more than 70 
percent have said that such information has 
influenced a decision about treatment, demonstrates 
the power of online applications to influence, and 
ultimately improve health care (Davis & Farrell, 
2001). Health educators need to do their part to 
insure that the information they provide is not only 
reliable but also comprehendible. 
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